Papers with Massive Text Embedding Benchmark
The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design (2025.naacl-long)
Copied to clipboard
| Challenge: | Embedding models are used in tasks such as information retrieval and semantic textual similarity. |
| Approach: | They propose a new Russian-focused embedding model called ru-en-RoSBERTa and a benchmark for Russian language . they propose to use the roMTEB benchmark to assess Russian and multilingual models . |
| Outcome: | The proposed model achieves results that are on par with state-of-the-art models in Russian. |
ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing frameworks for large language model embeddings have limited support for only a limited range of architectures and fine-tuning strategies. |
| Approach: | They propose a framework that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
| Outcome: | The proposed framework enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
VN-MTEB: Vietnamese Massive Text Embedding Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | a lack of large-scale test datasets makes it difficult to evaluate AI models before deploying them in real-world projects. |
| Approach: | They propose a Vietnamese benchmark for embedding models that leverages large language models and embeddable models to translate and filter samples from the Massive Multilingual Text Embedding Benchmark. |
| Outcome: | The proposed benchmark outperforms existing models in Vietnamese and English tasks with 41 datasets. |
PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of sentence embedding models rely on static tests like the Massive Text Embedding Benchmark (MTEB) repeated tuning on a fixed suite can inflate reported performance and obscure real-world robustness. |
| Approach: | They propose a dynamic protocol that generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
| Outcome: | The proposed protocol generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
MTEB: Massive Text Embedding Benchmark (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing text embeddings are evaluated on a small set of datasets, not covering their possible applications to other tasks. |
| Approach: | They propose a benchmarking framework that evaluates 8 embedding tasks covering 58 datasets and 112 languages. |
| Outcome: | The proposed model is the most comprehensive benchmark of text embeddings to date. |
Training LLMs to be Better Text Embedders through Bidirectional Reconstruction (2025.emnlp-main)
Copied to clipboard
Chang Su, Dengliang Shi, Siyuan Huang, Jintao Du, Changhua Meng, Yu Cheng, Weiqiang Wang, Zhouhan Lin
| Challenge: | Existing text embedding approaches often leverage the embeddment of the final token, typically a reserved special token such as ‘[EOS]‘. |
| Approach: | They propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. |
| Outcome: | The proposed training stage improves performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
GASE: Generatively Augmented Sentence Encoding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarizing, or extracting keywords, followed by pooling the original and synthetic embeddings. |
| Approach: | They propose a training-free approach to improve sentence embeddings by applying generative text models for data augmentation at inference time. |
| Outcome: | The proposed approach does not require access to model parameters or computational resources typically required for fine-tuning state-of-the-art models. |
FaMTEB: Massive Text Embedding Benchmark in Persian Language (2025.findings-emnlp)
Copied to clipboard
Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, Arash Amini
| Challenge: | a comprehensive benchmark for Persian text embeddings is built upon the Massive Text Embedding Benchmark (MTEB) 63 datasets are included in the benchmark, including a novel task of summary retrieval. |
| Approach: | They propose a benchmark for Persian (Farsi) text embeddings built upon the Massive Text Embedding Benchmark. |
| Outcome: | The proposed framework includes 63 datasets spanning seven different tasks . the evaluation datasets were rigorously evaluated by humans and automated systems . |
Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies use LoRA to fine-tune existing LLMs, but this is limited by the data and training gap between them and embedding models. |
| Approach: | They propose a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder that integrates embeddings across different languages. |
| Outcome: | The proposed model improves performance on the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025). |
Enhancing Lexicon-Based Text Embeddings with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. |
| Approach: | They introduce the first lexicon-based embeddings that consolidates the vocabulary space through token embeddation clustering to handle the issue of token redundancy in LLM vocabularies. |
| Outcome: | The proposed model outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) it also supports efficient dimension pruning without any specialized objectives like Matryoshka Representation Learning. |
MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in embedding resources have led to a lack of representation of the Dutch language in multilingual resources. |
| Approach: | They introduce Massive Text Embedding Benchmark for Dutch (MTEB-NL) which includes existing Dutch datasets and newly created ones, covering a wide range of tasks. |
| Outcome: | The proposed models demonstrate strong performance across multiple tasks. |